Layered Scene Decomposition CODEC System and Methods

ABSTRACT

A system and methods for a CODEC driving a real-time light field display for multi-dimensional video streaming, interactive gaming and other light field display applications is provided applying a layered scene decomposition strategy. Multi-dimensional scene data is divided into a plurality of data layers of increasing depths as the distance between a given layer and the plane of the display increases. Data layers are sampled using a plenoptic sampling scheme and rendered using hybrid rendering, such as perspective and oblique rendering, to encode light fields corresponding to each data layer. The resulting compressed, (layered) core representation of the multi-dimensional scene data is produced at predictable rates, reconstructed and merged at the light field display in real-time by applying view synthesis protocols, including edge adaptive interpolation, to reconstruct pixel arrays in stages (e.g. columns then rows) from reference elemental images.

CLAIM OF PRIORITY

This patent is a continuation of and claims the benefit of allowed U.S.Non-Provisional application Ser. No. 15/683,992 filed Aug. 23, 2017,published as US2019/0068973 on Feb. 28, 2019, the entire contents ofwhich is hereby expressly incorporated by reference herein in itsentirety.

FIELD OF THE INVENTION

The present disclosure relates to image (light-field) data encoding anddecoding, including data compression and decompression systems andmethods for the provision of interactive multi-dimensional content at alight field display.

BACKGROUND OF THE INVENTION

Autostereoscopic, high-angular resolution, wide field of view (FOV),multi-view displays provide users with an improved visual experience. Athree dimensional display that can pass the 3D Turing Test (described byBanks et al.) will require a light field representation in place of thetwo dimensional images projected by standard existing displays. Arealistic light field representation requires enormous amounts ofbandwidth to transmit the display data, which will comprise at leastgigapixels of data. These bandwidth requirements currently exceed thebandwidth capabilities provided by technologies previously known in theart; the upcoming consumer video standard is 8K Ultra High-Def (UHD),which provides only 33.1 megapixels of data per display.

Compressing data for transmission is previously known in the art. Datamay be compressed for various types of transmission, such as, but notlimited to: long-distance transmission of data over internet or ethernetnetworks; or transmission of a synthetic multiple-view created by agraphical processing unit (GPU) and transferred to a display device.Such data may be used for video streaming, real-time interactive gaming,or any other light-field display.

Several CODECS for compressed light-field transmission are previouslyknown in the art. Olsson et al. teach compression techniques where anentire light-field data set is processed to reduce redundancy andproduce a compressed representation. Subcomponents (i.e., elementalimages) of the light field are treated as a video sequence to exploitredundancy using standard video coding techniques. Vetro et al. teachmultiple-view specializations of compression standards that exploitredundancy between the light field subcomponents to achieve bettercompression rates, but at the expense of more intensive processing.These techniques may not achieve a sufficient compression ratio, andwhen a good ratio is achieved the encoding and decoding processes arebeyond real-time rates. These approaches assume that the entire lightfield exists in a storage disk or memory before being encoded. Thereforelarge light-field displays requiring large numbers of pixels introduceexcessive latency when reading from a storage medium.

In an attempt to overcome hardware limitations for the delivery ofmulti-dimensional content in real-time, various methods and systems areknown, however, these methods and systems present their own limitations.

U.S. Pat. No. 9,727,970 discloses a distributed, in parallel(multi-processor) computing method and apparatus for generating ahologram by separating 3D image data into data groups, calculating fromthe data groups hologram values to be displayed at different positionson the holographic plane and summing the values for each position forgenerating a holographic display. As a disclosure focused on generatinga holographic display, the strategies applied involve manipulating fineat a smaller scale than light field and in this instance ischaracterized by the sorting and dividing of data according to colour,followed by colour image planes and then further dividing the planeimages into sub-images.

US Patent Publication No. 20170142427 describes content adaptive lightfield compression based on the collapsing of multiple elemental images(hogels) into a single hogel. The disclosure describes achieving aguaranteed compression rate, however, image lossiness varies and incombining hogels as disclosed there is no guarantee of redundancy thatcan be exploited.

US Patent Publication No. 20160360177 describes methods for fullparallax compressed light field synthesis utilizing depth informationand relates to the application of view synthesis methods for creating alight field from a set of elemental images that form a subset of a totalset of elemental images. The view synthesis techniques described hereindo not describe or give methods to handle reconstruction artifactscaused during backwards warping.

US Patent Publication No. 20150201176 describes methods for fullparallax compressed light field 3D imaging systems disclosing thesubsampling of elemental images in a light field based on the distanceof the objects in the scene being captured. Though the methods describethe possibility of downsampling the light field using simple conditionsthat could enhance the speed of encoding, in the worse case 3D scenesexist where no down-sampling would occur and the encoding would fallback on transform encoding techniques which rely on having the entirelight field to exist prior to encoding.

There remains a need for increased data transmission capabilities,improved data encoder-decoders (CODECs), and methods to achieve bothimproved data transmission and CODEC capabilities for the real-timedelivery of multi-dimensional content to a light field display.

SUMMARY OF THE INVENTION

The present invention relates generally to 3D image data encoding anddecoding for driving a light field display in real-time, which overcomesor can be implemented with present hardware limitations.

It is an object of the present disclosure to provide a CODEC withreduced system transmission latency and high bandwidth rates to providefor the production of a light field, in real time, with good resolution,at a light field display, for application in video streaming, andreal-time interactive gaming. Light field or 3D scene data isdeconstructed into layers (corresponding to layered light fields),sampled and rendered to compress the data for transmission and thendecoded to construct and merge light fields corresponding to the datalayers at a light field display.

According to one aspect there is provided a computer-implemented methodcomprising the steps of:

i. receiving from a source a data set comprising light field data or a3D description of a scene;ii. partitioning said data set into a plurality of data layers, eachdata layer comprising elemental image content; andiii. sampling and rendering each data layer of the plurality of datalayers to encode a light field corresponding to each data layer andproduce a set of compressed data encoding a set of light fields.

In one embodiment of the method, the partitioning of the data set into aplurality of data layers is implemented according to a layering schemeto facilitate the encoding of the set of light fields as a distributionof light fields arranged such that the depth of each light fieldincreases as the distance between said light field and a display surfaceincreases. In a related embodiment, the depth of each light fieldincreases exponentially.

In another embodiment of the method, the sampling of each data layer ofsaid plurality of data layers is implemented according to one or moreplenoptic sampling schemes specifying, for each light fieldcorresponding to each data layer, reference elemental images sampledfrom the elemental image content of each data layer.

In still a further embodiment of the method, the rendering of a datalayer is executed by applying a rendering technique selected toefficiently encode the light field corresponding to said data layer. Inrelated embodiments, the rendering technique selected may be aperspective rendering, oblique rendering, or ray casting technique.

In one embodiment, the method further comprises the step of assigning avariable fractional bit representation to each data layer of theplurality of data layers to optimize compression of the data set.

In yet another embodiment, the method further comprises the steps oftransmitting and decoding the set of compressed data to construct theset of light fields. In one related embodiment, the step of transmittingthe compressed data is performed using a 2D display cable.

In a further related embodiment of the method, the step of decoding theset of compressed data is implemented by executing a view synthesisprotocol for each data layer sampled and rendered to encode the lightfield corresponding to each data layer. In other related embodiments,each view synthesis protocol comprises instructions the multi-stagereconstruction of a pixel array from reference elemental images used toconstruct the light field corresponding to each data layer. Themulti-stage reconstruction of the pixel array is performed in a firststage by column decoding and in a second stage by row decoding, oralternatively in a first stage by row decoding and in a second stage bycolumn decoding.

In still another embodiment of the method, one or more of the viewsynthesis protocols is executed with the application of edge adaptiveinterpolation to optimize the reconstruction of the pixel array.

In one embodiment, the method further comprises the step of merging thelight fields in the set of light fields to produce a display light fieldat a display. To produce the display light field, the set of compresseddata is used to produce a core encoded representation of the displaylight field. In a related embodiment, method still further comprises thesteps of encoding and compressing a residue encoded representation ofthe display light field. The core and residue encoded representations ofthe set of light fields can be used to produce the display light field.In illustrative embodiments, the display light field represents afrustum volume of the display, or alternatively an inner frustum and anouter frustum volume of the display.

According to another aspect there is provided a computer-implementedsystem comprising:

a source for a data set a data set comprising light field data or a 3Ddescription of a scene;an encoder in communication with said source, comprising one or moreprocessors configured to:

-   -   i. partition the data set received from said source into a        plurality of data layers, each data layer comprising elemental        image content; and    -   ii. sample and render each data layer of the plurality of data        layers to encode a light field corresponding to each data layer        and produce a set of compressed data encoding a set of light        fields;        a means for transmitting the set of compressed data from the        encoder to a decoder; and        a decoder comprising one or more processors configured to decode        the set of compressed data to construct a set of light fields.

In one embodiment of the system, the one or more processors of theencoder partition the data set into a plurality of data layers byimplementing a layering scheme to facilitate the encoding of the set oflight fields as a distribution of light fields arranged such that thedepth of each light field increases as the distance between said lightfield and a display surface increases. In a related embodiment, thedepth of each light field increases exponentially.

In another embodiment of the system, the one or more processors of theencoder sample of each data layer of said plurality of data layers byimplementing one or more plenoptic sampling schemes specifying, for eachlight field corresponding to each data layer, reference elemental imagessampled from the elemental image content of each data layer.

In still a further embodiment of the system, the one or more processorsof the encoder render each data layer by executing by applying arendering technique selected to efficiently encode the light fieldcorresponding to said data layer. In related embodiments the renderingtechnique selected may be a perspective rendering, oblique rendering, orray casting technique.

In one embodiment of the system, the one or more processors of theencoder are further configured to assign a variable fractional bitrepresentation to each data layer of the plurality of data layers tooptimize compression of the data set.

In yet another embodiment of the system, the transmission means is a 2Ddisplay cable.

In a further related embodiment of the system, the one or moreprocessors of the decoder implement the decoding of the set ofcompressed data by executing a view synthesis protocol for each datalayer sampled and rendered to encode the light field corresponding toeach data layer. In other related embodiments, each view synthesisprotocol comprises instructions the multi-stage reconstruction of apixel array from reference elemental images used to construct the lightfield corresponding to each data layer. The multi-stage reconstructionof the pixel array is performed in a first stage by column decoding andin a second stage by row decoding, or alternatively in a first stage byrow decoding and in a second stage by column decoding.

In still another embodiment of the system, one or more of the viewsynthesis protocols is executed with the application of edge adaptiveinterpolation to optimize the reconstruction of the pixel array.

In one embodiment of the system, the one or more processors of thedecoder are further configured to merge the light fields in the set oflight fields to produce a display light field at a display. To producethe display light field, the set of compressed data is used to produce acore encoded representation of the display light field. In a relatedembodiment, method still further comprises the steps of encoding andcompressing a residue encoded representation of the display light field.The core and residue encoded representations of the set of light fieldscan be used to produce the display light field. In illustrativeembodiments, the display light field represents a frustum volume of thedisplay, or alternatively an inner frustum and an outer frustum volumeof the display.

According to a further aspect, there is provided a computer-implementedsystem comprising one or more processors and one or more memorycomponents comprising instructions configured to cause the one or moreprocessors to perform a method comprising the steps of:

-   -   i. receiving from a source a data set comprising light field        data or a 3D description of a scene;    -   ii. partitioning said data set into a plurality of data layers,        each data layer comprising elemental image content; and    -   iii. sampling and rendering each data layer of the plurality of        data layers to encode a light field corresponding to each data        layer and produce a set of compressed data encoding a set of        light fields.

According to yet another aspect, there is provided a non-transitory,computer readable medium comprising instructions configured to cause theone or more processors to perform a method comprising the steps of:

-   -   i. receiving from a source a data set comprising light field        data or a 3D description of a scene;    -   ii. partitioning said data set into a plurality of data layers,        each data layer comprising elemental image content; and    -   iii. sampling and rendering each data layer of the plurality of        data layers to encode a light field corresponding to each data        layer and produce a set of compressed data encoding a set of        light fields.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features of the invention will become more apparent inthe following detailed description in which reference is made to theappended drawings.

FIG. 1: is a schematic representation (block diagram) of an embodimentof a layered scene decomposition (CODEC) system according to the presentdisclosure.

FIG. 2: is a schematic top-down view of the inner frustum volume andouter frustum volume of a light field display.

FIG. 3A: illustrates schematically the application of edge adaptiveinterpolation for pixel reconstruction according to the presentdisclosure.

FIG. 3B: illustrates a process flow for reconstructing a pixel array.

FIG. 4: illustrates schematically elemental images specified by asampling scheme within a pixel matrix, as part of the image (pixel)reconstruction process according to the present disclosure.

FIG. 5: illustrates schematically a column-wise reconstruction of apixel matrix, as part of the image (pixel) reconstruction processaccording to the present disclosure.

FIG. 6: illustrates a subsequent row-wise reconstruction of the pixelmatrix, as part of the image (pixel) reconstruction process according tothe present disclosure.

FIG. 7: illustrates schematically an exemplary CODEC system embodimentaccording to the present disclosure.

FIG. 8: illustrates schematically an exemplary layered scenedecomposition of an image data set (a layering scheme of ten layers)correlating to the inner frustum light field of a display.

FIG. 9: illustrates schematically an exemplary layered scenedecomposition of image data (two layering schemes of ten layers)correlating to the inner frustum and outer frustum light field regions,respectively, of a display.

FIG. 10: illustrates an exemplary CODEC process flow according to thepresent disclosure.

FIG. 11: illustrates an exemplary process flow for encoding 3D image(scene) data to produce layered and compressed core encoded (lightfield) representations, according to the present disclosure.

FIG. 12: illustrates an exemplary process flow for decoding core encodedrepresentations to construct a (display) light field at a display,according to the present disclosure.

FIG. 13: illustrates an exemplary process flow for encoding and decodingresidue image data for use with core image data to produce a(display/final) light field at a display according to the presentdisclosure.

DETAILED DESCRIPTION OF THE INVENTION

The present invention relates generally to CODEC systems and methods forlight field data or multi-dimensional scene data compression anddecompression to provide for the efficient (rapid) transmission andreconstruction of a light field at a light field display.

Various features of the invention will become apparent from thefollowing detailed description taken together with the illustrations inthe Figures. The design factors, construction and use of the layeredscene decomposition CODEC disclosed herein are described with referenceto various examples representing embodiments which are not intended tolimit the scope of the invention as described and claimed herein. Theskilled technician in the field to which the invention pertains willappreciate that there may be other variations, examples and embodimentsof the invention not disclosed herein that may be practiced according tothe teachings of the present disclosure without departing from the scopeand spirit of the invention.

Definitions

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention pertains.

The use of the word “a” or “an” when used herein in conjunction with theterm “comprising” may mean “one,” but it is also consistent with themeaning of “one or more,” “at least one” and “one or more than one.”

As used herein, the terms “comprising,” “having,” “including” and“containing,” and grammatical variations thereof, are inclusive oropen-ended and do not exclude additional, unrecited elements and/ormethod steps. The term “consisting essentially of” when used herein inconnection with a composition, device, article, system, use or method,denotes that additional elements and/or method steps may be present, butthat these additions do not materially affect the manner in which therecited composition, device, article, system, method or use functions.The term “consisting of” when used herein in connection with acomposition, device, article, system, use or method, excludes thepresence of additional elements and/or method steps. A composition,device, article, system, use or method described herein as comprisingcertain elements and/or steps may also, in certain embodiments consistessentially of those elements and/or steps, and in other embodimentsconsist of those elements and/or steps, whether or not these embodimentsare specifically referred to.

As used herein, the term “about” refers to an approximately +/−10%variation from a given value. It is to be understood that such avariation is always included in any given value provided herein, whetheror not it is specifically referred to.

The recitation of ranges herein is intended to convey both the rangesand individual values falling within the ranges, to the same place valueas the numerals used to denote the range, unless otherwise indicatedherein.

The use of any examples or exemplary language, e.g. “such as”,“exemplary embodiment”, “illustrative embodiment” and “for example” isintended to illustrate or denote aspects, embodiments, variations,elements or features relating to the invention and not intended to limitthe scope of the invention.

As used herein, the terms “connect” and “connected” refer to any director indirect physical association between elements or features of thepresent disclosure. Accordingly, these terms may be understood to denoteelements or features that are partly or completely contained within oneanother, attached, coupled, disposed on, joined together, incommunication with, operatively associated with, etc., even if there areother elements or features intervening between the elements or featuresdescribed as being connected.

As used herein, the term “pixel” refers to a light source and lightemission mechanism used to create a display.

As used herein, the term “light field” at a fundamental level refers toa function describing the amount of light flowing in every directionthrough points in space, free of occlusions. Therefore, a light fieldrepresents radiance as a function of position and direction of light infree space. A light field can be synthetically generated through variousrendering processes or may be captured from a light field camera or froman array of light field cameras.

As used herein, the term “light field display” is a device whichreconstructs a light field from a finite number of light field radiancesamples input to the device. The radiance samples represent the colorcomponents red, green and blue (RGB). For reconstruction in a lightfield display, a light field can also be understood as a mapping from afour dimensional space to a single RGB color. The four dimensionsinclude the vertical and horizontal dimensions of the display and twodimensions describing the directional components of the light field. Alight field is defined as the function:

LF:(x,y,u,v)→(r,g,b)

For a fixed x_(f), y_(f), LF(x_(f), y_(f), u, v) represents a twodimensional (2D) image referred to as an “elemental image”. Theelemental image is a directional image of the light field from the fixedx_(f), y_(f) position. When a plurality of elemental images areconnected side by side, the resulting image is referred to as an“integral image”. The integral image can be understood as the entirelight field required for the light field display.

It is contemplated that any embodiment of the compositions, devices,articles, methods and uses disclosed herein can be implemented by oneskilled in the art, as is, or by making such variations or equivalentswithout departing from the scope and spirit of the invention.

Layered Scene Decomposition (LSD) CODEC System and Methods

The CODEC according to the present disclosure applies a strategy ofdrawing upon known sampling, rendering, and view synthesis methods forgenerating light field displays, adapting said strategies for use inconjunction with a novel layered scene decomposition strategy asdisclosed herein, including its derivation, implementation andapplications.

3D Displays

A conventional display as previously known in the art consists ofspatial pixels substantially evenly-spaced and organized in atwo-dimensional row, allowing for an idealized uniform sampling. Bycontrast, a three-dimensional display requires both spatial and angularsamples. While the spatial sampling of a typical three-dimensionaldisplay remains uniform, the angular samples cannot necessarily beconsidered uniform in terms of the display's footprint in angular space.For a review of various light field parameterizations for angular raydistributions, please see U.S. Pat. No. 6,549,308.

The angular samples, also known as directional components of the lightfield, can be parameterized in various ways, such as the planarparameterizations taught by Gortler et. al in “The Lumigraph”. When thelight field function is discretized in terms of position, the lightfield can be understood as a regularly-spaced array ofplanar-parameterized pinhole projectors, as taught by Chai in “PlenopticSampling”. For a fixed x_(f), y_(f) the elemental image LF(x_(f), y_(f),u, v) represents a two dimensional image which may be understood as animage projected by a pinhole projector with an arbitrary rayparameterization. For a light field display, the continuous elementalimage is represented by a finite number of light field radiance samples.For an idealized, planar parameterized pinhole projector, said finitenumber of samples are mapped into the image plane as a regularly-spacedarray (the regular spacing within the plane does not correspond to aregular spacing in the corresponding angular directional space).

The consideration of planar parameterizations is not intended to limitthe scope or spirit of the present disclosure, as the directionalcomponents of the light field can be parameterized by a variety of otherarbitrary parameterizations. For example, lens distortions or otheroptical effects in a physically embodied pinhole projector can bemodeled as distortions of the planar parameterization. In addition,display components may be defined through a warping function, such astaught by Clark et al. in “A transformation method for thereconstruction of functions from nonuniformly spaced samples.”

A warping function α(u, v) defines a distorted planar parameterizationof the pinhole projector, producing arbitrary alternate angulardistributions of directional rays in the light field. The angulardistribution of rays propagating from a light field pinhole projector isdetermined by the pinhole projector's focal length f and a correspondingtwo dimensional warping function α(u, v).

An autostereoscopic light field display projecting a light field for oneor more users is defined as:

D=(M _(x) ,M _(y) ,N _(u) ,N _(v) ,f,α,D _(LP))

Where (M_(x), M_(y)) are the horizontal and vertical dimensions of thedisplay's spatial resolution and (N_(u), N_(v)) are the horizontal andvertical dimensions of the display's angular resolution components. Thedisplay is an array of idealized light field projectors, with pitchD_(LP), focal length f, and a warping function α defining thedistribution of ray directions for the light field projected by thedisplay.

A light field LF(x, y, u, v) driving a light field display D=(M_(x),M_(y), N_(u), N_(v), f, α, D_(LP)) requires M_(x) light field radiancesamples in the x direction, M_(y) light field radiance samples in the ydirection, and N_(u), and N_(v) light field radiance samples in the uand v directions. While D is defined with a single warping function α,each of the light field planar-parameterized pinhole projectors withinthe array of idealized light field pinhole projectors may have a uniquewarping function α, if significant microlens variations exist in apractical pinhole projector causing the angular ray distribution to varysignificantly from one microlens to another microlens.

Light Field Display Rendering

In “Fast computer graphics rendering for full parallax spatialdisplays,” Halle et al. provide a method for rendering objects locatedwithin an inner frustum volume and outer frustum volume of the display.FIG. 2 illustrates a light field display representing objects within avolumetric region defined by these two separate viewing frusta, with theinner frustum volume (110) located behind the display surface (300)(i.e., within the display) and the outer frustum volume (210) located infront of the display surface (i.e. outside of the display). Asillustrated, various objects (shown schematically as prismatic andcircular shapes) are located at varying depths from the display surface(300).

Halle et al. teach a double frustum rendering technique, where the innerfrustum volume and outer frustum volume are separately rendered as twodistinct light fields. The inner frustum volume LF_(O), (x, y, u, v) andouter frustum volume LF_(P)(x, y, u, v) are recombined into the singlelight field LF(x, y, u, v) through a depth merging process.

The technique uses a pinhole camera rendering model to generate theindividual elemental images of the light field. Each elemental image(i.e. each rendered planar-parameterized pinhole projector image)requires the use of two cameras: one camera to capture the inner frustumvolume and one camera to capture the outer frustum volume. Halle et al.teach rendering a pinhole projector image at a sampling region of thelight field using a standard orthoscopic camera and its conjugatepseudoscopic camera. For a pinhole camera C, the corresponding conjugatecamera is denoted as C⁺.

To capture an elemental image within a light field display withprojectors parameterized using warping function α, a generalized pinholecamera based on a re-parameterization of an idealizedplanarly-parameterized pinhole camera is used. As taught by Gortler etal., a pinhole camera C with a focal length f has light rays defined bya parameterization created by two parallel planes. Pinhole camera Ccaptures an image I_(C)(u, v), where (u, v) are coordinates in the rayparameterization plane. The generalized pinhole camera, C_(α), is basedupon a planar parameterized camera warped using a two dimensional,continuous, invertible time-warping function, as taught by Clark et al.With a warping function α(u, v,), the inverse is γ(u, v,). Therefore,the image of C_(α), I_(Cα)=I_(C) (α(u, v)).

Given the generalized pinhole camera, C_(α), a conjugate generalizedcamera C_(α)* is formed to complete double frustum rendering. The viewsgenerated from a M_(x)×M_(y) grid of generalized pinhole camera pairsare rendered to render the light field for the light field display.

Therefore, the set of all generalized pinhole camera pairs that must berendered to produce light field LF(x, y, u, v) for a given light fielddisplay D=(M_(x), M_(y), N_(u), N_(v), f, α, D_(LP)) is defined as:

{(C _(α) ,C _(α)*)(x,y)|1≤x≤M _(x),1≤y≤M _(y)}

A set of orthoscopic cameras (O={(C_(α)(x, y)|1≤x≤M_(x), 1≤y≤M_(y)})capture the light field image corresponding to the inner frustum volumeand a set of conjugate generalized cameras (P={(C_(α)*(x, y)|1≤x≤M_(x),1≤y≤M_(y)}) capture the image corresponding to the outer frustum volume.As described above, the inner frustum volume and outer frustum volumeare combined into a single light field.

Data Compression for Light Field Display

Piao et al. utilize a priori physical properties of a light field inorder to identify redundancies in the data. The redundancies are used todiscard elemental images based on the observation that elemental imagesrepresenting neighboring points in space contain significant overlappedinformation. This avoids performing computationally complex datatransforms in order to identify information to discard. Such methods donot utilize depth map information associated with each elemental image.

In “Compression for Full-Parallax Light Field Displays,” Graziosi et al.propose criteria to sub-sample elemental images based on simple pinholecamera coverage geometry to reduce light field redundancy. Thedownsampling technique taught by Graziosi et al. is simpler than thecomplicated basis decompositions often employed in other CODEC schemesfor two dimensional image and video data. Where an object is locateddeep within a scene, the light field is sampled at a smaller rate. Forexample, when two separate pinhole cameras provide two different fieldsof view, there is very little difference from one elemental image to thenext elemental image, and the fields of view from the two pinholecameras overlap. While the views are subsampled based on geometric(triangle) overlap, the pixels within the views are not compressed.Because these pixels can be substantial, Graziosi et al. compress thepixels with standard two-dimensional image compression techniques.

Graziosi et al. teach that the sampling gap (ΔEI) between elementalimages, based on the minimum depth of an object d, can be calculated asfollows, where θ represents the light field display's field of view andP represents the lens pitch of the integral imaging display:

${\Delta \; {EI}} = \frac{\left( {2d} \right){\tan \left( {\theta/2} \right)}}{P}$

This strategy provides a theoretically lossless compression forfronto-parallel planar surfaces when there are no image occlusions. Asshown in the formula, the sampling gap increases with d, providing animproved compression rate when fewer elemental images are required. Forsufficiently small d, ΔEI can reach 0. Therefore, this downsamplingtechnique gives no guaranteed compression rate. In a scene with multiplesmall objects, where the objects are close to the screen or are at thescreen distance, each elemental image would have at least some pixels ata 0 depth and this technique would provide no gains, i.e. ΔEI=0throughout the integral image.

Graziosi et al. equate the rendering process with the initial encodingprocess. Instead of producing all of the elemental images, this methodonly produces the number needed to reconstruct the light field whileminimizing any loss of information. Depth maps are included with theelemental images selected for encoding and the missing elemental imagesare reconstructed using well-established warping techniques associatedwith depth image-based rendering (DIBR). The selected elemental imagesare further compressed using methods similar to the H.264/AVC method,and the images are decompressed prior to the final DIBR-based decodingphase. While this method provides improved compression rates withreasonable signal distortion levels, no time-based performance resultsare presented. Such encoding and decoding cannot provide goodlow-latency performance for high bandwidth rates. In addition, thismethod is limited to use for a single object that is far away from thedisplay screen; in scenes with multiple overlapping objects and manyobjects close to the display screen, the compression would be forcedback to use H.264/AVC style encoding.

Chai teaches plenoptic sampling theory to determine the amount ofangular bandwidth required to represent fronto-parallel planar objectsat a particular scene depth. Zwicker et al. teach that the depth offield of a display is based on the angular resolution, with moreresolution providing a greater depth of field. Therefore, objects closeto the display screen are represented adequately with lower angularresolution, while far objects require larger angular resolutions.Zwicker et al. teach the maximum display depth of field with idealprojective lenses based on planar parameterization is:

$Z_{DOF} = \frac{{fP}_{l}}{P_{p}}$

where P_(l) is the lens pitch and P_(p) is the pixel pitch and f is thefocal length of the lenses. In a three dimensional display with anisotropic angular resolution (i.e. N=N_(u)=N_(v)), N=P_(l)/P_(p).Therefore, Z_(DOF)=fN.

To determine the angular resolution required to represent the fullspatial resolution of the display, at a given depth d, the equation isrearranged as:

${N_{res}(d)} = \frac{d}{f}$

Therefore, each focal length distance into the scene adds another pixelof angular resolution required to fully represent objects at the givenspatial resolution of the display screen.

Layered Scene Decomposition and Sampling Scheme

The sampling gap taught by Graziosi et al. and the plenoptic samplingtheory taught by Zwicker et al. provide complimentary light fieldsampling strategies: Graziosi et al. increase downsampling for distantobjects (ΔEI) while Zwicker et al. increase downsampling for nearobjects (N_(res)). However, when downsampling a single light fieldrepresenting a scene, the combination of these strategies does notguarantee compression. Therefore, the present disclosure divides amultiple-dimensional scene into a plurality of layers. This divisioninto a plurality of (data) layers is referred to herein as a layeredscene decomposition. Where K₁ and K₂ are natural numbers, we defineL=(K₁, K₂, L^(O), L^(P)), partitioning the inner and outer frustumvolumes of a three-dimensional display. The inner frustum is partitionedinto a set of K₁ layers, where L^(O)={l₁ ^(O), l₂ ^(O) . . . l_(K) ₁^(O)}. Each inner frustum layer is defined by a pair of bounding planesparallel to the display surface at distances d_(min)(l_(i) ^(O)) andd_(max)(l_(i) ^(O)) for 1≤i≤K₁ from the display surface plane. The outerfrustum is partitioned into a set of K₂ layers, where L^(P)={l₁ ^(P), l₁^(P) . . . l_(K) ₂ ^(O)}. Each outer frustum layer is defined by a pairof bounding planes parallel to the display surface at distancesd_(min)(l_(i) ^(O)) and d_(max)(l_(i) ^(P)) for 1≤i≤K₂ from the displaysurface plane. In alternate embodiments, the inner and outer frustumvolumes may be divided by layering schemes differing from each other.

Each of the layered scene decomposition layers has an associated lightfield (herein also referred to as a “light field layer”) based on thescene restrictions to the planar bounding regions of the layer. Considera layered scene decomposition L=(K₁, K₂, L^(O), L^(P)) for a light fielddisplay D=(M_(x), M_(y), N_(u), N_(v), f, α, D_(LP)) with an innerfrustum layer l_(i) ^(P)ϵL^(P) for 1≤i≤K₁, or an outer frustum layerl_(j) ^(O)ϵL^(O) for 1≤j≤K₂. The inner frustum light field

LF_(l_(i)^(O))

is generated from the set of generalized pinhole cameras O={C_(a) (x,y)|1≤x≤M_(x), 1≤y≤M_(y)}. This equation is constrained such that onlythe space at distance d from the light field display surface, whered_(min)(l_(i) ^(P))≤d≤d_(max)(l_(i) ^(P)), is imaged. Therefore, for aninner frustum layer with a fixed x, y and C_(α)(x, y)ϵO, we calculate

LF_(l_(i)^(O))(x, y, u, v) = I_(C_(a)(x, y)).

Similarly the outer frustum light field

LP_(l_(i)^(P))

is generated from the set of generalized pinhole cameras P={C*_(α)(x,y)|1≤x≤M_(x), 1≤y≤M_(y)}. This equation is constrained such that onlythe space at distance d from the light field display surface, whered_(min)(l_(i) ^(P))≤d≤d_(max)(l_(i) ^(P)), is imaged. Therefore, for anouter frustum layer with a fixed x, y and C_(α)(x, y)ϵP, we calculate

LP_(l_(i)^(P)) = I_(C_(a)(x, y)).

The sets of light fields for the inner and outer frustum regionsrelative to the layered scene decomposition L can be further defined.Assume a light field display D=(M_(x), M_(y), N_(u), N_(v), f, α,D_(LP)) with a layered scene decomposition L=(K₁, K₂, L^(O), L^(P)). Theset of inner frustum region light fields is defined as

O^(LF) = {LF_(l_(i)^(O))|1 ≤ i ≤ K₁}.

The set of outer frustum region light fields is defined as

P^(LF) = {LF_(l_(i)^(P))|1 ≤ i ≤ K₂}.

As defined, a layered scene decomposition generates a light field foreach layer. For any layered scene decomposition, orthoscopic camerasgenerate inner frustum volume light fields and pseudoscopic camerasgenerate outer frustum volume light fields. If a scene captured by thesegeneralized pinhole camera pairs is comprised of only opaque surfaces,each point of the light field has an associated depth value whichindicates the distance from the generalized pinhole camera plane to thecorresponding point in space imaged. When given a light field

LF_(l_(i)^(O))  ϵ  O^(LF)  or  LF_(l_(j)^(O))  ϵ  P^(LF),

the

LF_(l_(i)^(O))

depth map is formally defined as

D_(m)[LF_(l_(i)^(O))](x, y, u, v),

and the

LF_(l_(i)^(P))

depth map is formally defined as

D_(m)[LF_(l_(i)^(P))](x, y, u, v).

The depth map D_(m)=∞ where there are no surface intersection pointscorresponding to the associated imaging generalized pinhole camera rays.Across their domains,

d_(min)(l_(i)^(P)) ≤ D_(m)[LF_(l_(i)^(P))](x, y, u, v) ≤ d_(max)(l_(i)^(P))andd_(min)(l_(i)^(O)) ≤ D_(m)[LF_(l_(i)^(O))](x, y, u, v) ≤ d_(max)(l_(i)^(O)).

In other words, depth maps associated with a layered scene decompositionlayer's light field are bound by the depth bounds of the layer itself.

A merging operation can re-combine the layered scene decomposition layersets back into the inner and outer frustum volumes, or LF_(O) andLF_(P). The inner and outer frustum volume light fields are merged withthe merging operator *_(m). For example, when given two arbitrary lightfields, LF₁(x, y, u, v) and LF₂ (x, y, u, v), wherei=argmin_(jϵ{1,2})D_(m)[LF_(j)](x, y, u, v), *_(m) is defined as:

LF ₁(x,y,u,v)*_(m) LF ₂(x,y,u,v)=LF _(i)(x,y,u,v)

Therefore, LF_(O)(x, y, u, v) and LF_(P)(x, y, u, v) can be recoveredfrom the sets O^(LF) and P^(LF) by merging the light fields associatedwith the inner and outer frustum layers. For example:

LF_(O) = LF_(l₁^(O))*_(m)  LF_(l₂^(O))*_(m)  …  *_(m)  LF_(l_(K₁)^(O))LF_(P) = LF_(l₁^(P))*_(m)  LF_(l₂^(P))*_(m)  …  *_(m)  LF_(l_(K₁)^(P))

The present disclosure provides a layered scene decomposition operationand an inverse operation which merges the data to reverse saiddecomposition. Performing a layered scene decomposition with K layers isunderstood to create K times as many individual light fields. The valueof the layered scene decomposition is in the light fields induced by thelayers; these light field layers are more suitable for downsampling thanthe original total light field or the inner frustum volume or outerfrustum volume light fields, as the total data size required formultiple downsampled layered scene decomposition light field layers withan appropriate sampling scheme is significantly less than the size ofthe original light field.

The skilled technician in the field to which the invention pertains willappreciate that there are multiple types of sampling schemes that cansuccessfully sample a light field. The sampling scheme S provided is notintended to limit or depart from the scope and spirit of the invention,as other sampling schemes, such as specifying individual sampling ratesfor each elemental image in the layered scene decomposition layer lightfields, can be employed. Relatively simple sampling schemes can providean effective CODEC with greater sampling control, therefore the presentdisclosure provides a simple sampling scheme to illustrate thedisclosure without limiting or departing from the scope and spirit ofthe invention.

A light field sampling scheme provided according to the presentdisclosure represents a light field encoding method. Given a displayD=(M_(x), M_(y), N_(u), N_(v), f,α, D_(LP)) and a layered scenedecomposition L=(K₁, K₂, L^(O), L^(P)), the present disclosure providesa sampling scheme S associated with L as an M_(x)×M_(y) binary matrixM_(S)[l_(i)] associated with any layer l_(i) in L^(O) or L^(P) and amapping function R(l_(i)) to map each layer l_(i) to a pairR(l_(i))=(n_(x), n_(y)). A binary ({0,1}) entry in M_(S)[l_(i)] at(x_(m), y_(m)) indicates if the elemental image LF_(l) _(i) (x_(m),y_(m), u, v) is included in the sampling scheme: a (1) indicates LF_(l)_(i) (x_(m), y_(m), u, v) is included, and a (0) indicates LF_(l) _(i)(x_(m), y_(m), u, v) is not included. R(l_(i))=(n_(x), n_(y)) indicatesthe elemental images in light field LF_(l) _(i) are sampled at aresolution of n_(x)×n_(y).

The present disclosure also provides a layered scene decomposition lightfield encoding process that draws upon plenoptic sampling theory. Thefollowing description pertains to the inner frustum volume L^(O) of alayered scene decomposition L, but the outer frustum volume L^(P) may beencoded in a similar fashion.

For each I_(i)ϵL^(o), the depth map of the corresponding light fieldLF_(l) _(i) is restricted to d in the range d_(min)(l_(i)^(O))≤d≤)d_(max)(l_(i) ^(O)). Based on the sampling scheme presentedabove, the present disclosure creates a sampling scheme S using thefollowing equation to guide the creation of M_(S)[l_(i) ^(O)]:

${\Delta \; {{EI}\left( {d_{\min}\left( l_{i}^{O} \right)} \right)}} = \frac{\left( {2{d_{\min}\left( l_{i}^{O} \right)}} \right){\tan \left( {\theta/2} \right)}}{D_{LP}}$

In other words, ΔEI guides the distance between “1” entries in the M_(S)matrix associated with each layered scene decomposition layer. Thefollowing equation sets the resolution of the individual elementalimages

${{R\left( l_{i}^{O} \right)} = \frac{d_{\max {(l_{i}^{O})}}}{f}},\frac{d_{\max {(l_{i}^{O})}}}{f}$

in a layer:

${N_{res}\left( {d_{\max}\left( l_{i}^{O} \right)} \right)} = \frac{d_{\max}\left( l_{i}^{O} \right)}{f}$

This sampling scheme, using both ΔEI and N_(res) to drive individuallayered scene decomposition layer sampling rates, can be considered as alayered plenoptic sampling theory sampling scheme (otherwise referred toherein as “plenoptic sampling scheme”). This plenoptic sampling schemeis based on a display utilizing the plenoptic sampling theory identityfunction α(t)=t. This per-layer sampling scheme provides losslesscompression for fronto-parallel planar scene objects where the objectswithin a layer do not occlude each other.

The assumption of only fronto-parallel planar scene objects isrestrictive and does not represent typical scenes; inevitably there areintra-layer occlusions, especially for layered scene decompositionlayers that are larger in size. To capture and encode a full range ofpotential scenes without introducing significant perceivable artifacts,the system can draw upon information in addition to the light fieldplenoptic sampling scheme of the present disclosure.

For example, surfaces are locally approximated by planar surfaces atvarious slanting angles. In “On the bandwidth of the plenopticfunction,” Do et al., theorize time-warping techniques allowing for thespectral characterization of slanted light field display planes. Thiswork suggests that a necessary decrease in downsampling and the need forprecise characterization of local bandwidth changes is induced by thedegree of surface slanting, the depth of objects in the scene, and thepositioning of objects at the FOV edge. Therefore, if signal distortionsfrom fronto-parallel geometry deviations are perceptually significant,residue representations can adaptively send additional or supplementalelemental image data (dynamically altering the static sampling scheme)to compensate for losses incurred.

The present disclosure therefore provides for the identification as“core” or “residue” information for the encoding and decoding of thelight field by the CODEC. When given a light field display D and acorresponding layered scene decomposition L with an associated samplingscheme S, the present disclosure considers the encoded, downsampledlight fields associated with L and S, as well as the number of layeredscene decomposition layers and the depth of said layers, as the “core”representation of a light field encoded and decoded by the CODEC. Anyadditional information transmitted along with the core (encoded)representation of the light field that may be required during thedecoding process is considered as the “residue” representation of thelight field to be processed by the CODEC and used together with the corerepresentation of the light field to produce the final light fielddisplayed.

Layer-Based Compression Analysis

Predictable compression rates are required to create a real-timerendering and transmission system, together with downsampling criteria(which do not indicate achievable compression rates). The followingprovides a compression analysis of the present disclosure's layeredscene decomposition encoding strategy.

As already described, downsampling a light field based on plenopticsampling theory alone does not offer guaranteed compression rates. Thepresent disclosure provides a downsampling light field encodingstrategy, allowing for a low-latency, real-time light field CODEC. Inone embodiment, complementary sampling schemes based on plenopticsampling theory, using both ΔEI and N_(res) are employed to driveindividual layered scene decomposition layer sampling rates. The layeredscene decomposition, representing the total 3D scene as a plurality oflight fields, expands the scene representation by a factor of the numberof layers. The present disclosure further contemplates that when layerdepths are chosen appropriately, compression rates can be guaranteedwhen combined with plenoptic sampling theory based downsampling.

For a light field LF_(l) _(i) corresponding to a given layered scenedecomposition layer l_(i), the layer's restricted depth range provides aguaranteed compression rate for the layer's light field. The achievablecompression ratio from downsampling a scene completely contained withina single layer can be explained in the following theorem:

-   -   Theorem 1    -   Consider a display D=(M_(x), M_(y), N_(u), N_(v), f,α, D_(LP))        with an isotropic angular resolution N=N_(u)=N_(v), a layered        scene decomposition L and an associated sampling scheme        S=(M_(s), R). Assume a layered scene decomposition layer l_(i)        with the corresponding light field LF_(l) _(i) such that        d_(min)(l_(i))<Z_(DOF)(D), and M_(s)[LF_(l) _(i) ] is selected        so the distance between “1” entries is set to ΔEI        (d_(min)(l_(i))) and R(l_(i))=N_(res)(d_(max)(l_(i))). The        compression ratio associated with S relative to the layered        scene decomposition layer l_(t) is

$1\text{:}{{N^{2}\left( \frac{d_{\min}\left( l_{i} \right)}{d_{\min}\left( l_{i} \right)} \right)}.}$

-   -   Proof 1    -   Consider a layered scene decomposition layer within the maximum        depth of field of the display, where

${d_{\min}\left( l_{i} \right)} = {{\frac{z_{DOF}}{c}\mspace{14mu} {and}\mspace{14mu} {d_{\max}\left( l_{i} \right)}} = \frac{z_{DOF}}{d}}$

for 0<c, d≤Z_(DOF). Therefore,

$c = {{\frac{z_{DOF}}{d_{\min}\left( l_{i} \right)}\mspace{14mu} {and}\mspace{14mu} d} = {{\frac{z_{DOF}}{d_{\max}\left( l_{i} \right)}\mspace{14mu} {and}\mspace{14mu} {d/c}} = {\frac{d_{\min}\left( l_{i} \right)}{d_{\max}\left( l_{i} \right)}.}}}$

Therefore ΔEI(d_(min)(l_(i))=N/c and N_(res)(d_(max)(l_(i)))=N/d.

-   -   Based on this rate of sub-sampling, the system requires every        (N/c)^(th) elemental image, therefore providing a compression        ratio of 1:(N/c)². The elemental image sub-sampling provides a        1:d² compression ratio. Therefore the total compression ratio is        1:(N/c)²*1:d²=1:N² (d/c)². The compression factor term

$c_{f} = \frac{d_{\min}\left( l_{i} \right)}{d_{\max}\left( l_{i} \right)}$

determines the compression ratio.

There may be an alternate case where d_(min)(l_(i))=Z_(DOF) and(d_(max)(l_(i))) can extend to any arbitrary depth. We know ΔEI(Z_(DOF))=N and N_(res) attains the maximum possible value of N for alldepths d≥Z_(DOF). Based on this rate of sub-sampling, the systemrequires every N^(th) elemental image, thus providing the light fieldwith a 1:N² compression ratio. Adding additional layered scenedecomposition layers beyond Z_(DOF) adds redundant representationalcapability when representing fronto-parallel planar objects. Therefore,when creating a core encoded representation, the total scene may beoptimally decomposed with the maximum depth of field in the layers.

Given the compression calculation expression for downsampling a layeredscene decomposition layer, we can determine how the compression factorvaries as the layer parameters vary. For a layer of a fixed width, ord_(max)(l_(i))−d_(min)(l_(i))=w for some w, the c_(f) term is minimizedwhen d_(max)(l_(i))−d_(min)(l_(i)) is closest to the display plane.Therefore, layered scene decomposition layers located closer to thedisplay plane require a narrower width to achieve the same compressionratio as layers located further away from the display plane. Thiscompression rate analysis can extend to scenes that are partitioned intomultiple adjacent fronto-planar layers located in the space from thedisplay plane until the depth Z_(DOF).

-   -   Theorem 2

Consider a display D=(M_(r), M_(y), N_(u), N_(v), f, a, D_(LP)) with anisotropic angular resolution N=N_(u)=N_(v), a layered scenedecomposition L and an associated sampling scheme S=(M_(s), R). LetS_(LF)=M_(r)M_(Y)N_(u)N_(v), denoting the number of image pixels in thelight field. The compression ratio of the layered scene decompositionrepresentation can be defined as:

$\frac{A}{S_{LF}} = {{\left( {1/N^{2}} \right){\sum_{i = 1}^{K}\left( {1/{c_{f}(i)}^{2}} \right)}} = {\left( {1/N^{2}} \right){\sum_{i = 1}^{K}\left( \frac{d_{\max}\left( l_{i} \right)}{d_{m\; i\; n}\left( l_{i} \right)} \right)^{2}}}}$

-   -   Proof 2    -   For a given layered scene decomposition layer downsampled with        compression ratio:

${S_{layer}(i)} = {\left( \frac{1}{N^{2}{c_{f}(i)}^{2}} \right)S_{LF}}$

-   -   To calculate the compression ratio, the size of each layer in        the compressed form is computed and summed, and the total        compressed layer size is divided by the size of the light field.        Consider a sum where the size of the compressed set of layers        is:

$A = {\sum_{i = 1}^{K}{\left( \frac{1}{N^{2}{c_{f}(i)}^{2}} \right)S_{LF}}}$

-   -   Therefore the compression ratio of the combined layers is:

$\frac{A}{S_{LF}} = {{\left( {1/N^{2}} \right)\mspace{11mu} {\sum\limits_{i = 1}^{K}\left( {1/{c_{f}(i)}^{2}} \right)}} = {\left( {1/N^{2}} \right){\sum\limits_{i = 1}^{K}\left( \frac{f + {i\; \Delta \; L}}{f + {\left( {i - 1} \right)\; \Delta \; L}} \right)^{2}}}}$

-   -   -   In a system where the layered scene decomposition layers are            of variable width, with d_(min)(i) and d_(max)(i)            representing the front and back boundary depths of the            i^(th) layer, the compression ratio of the layered scene            decomposition representation is:

$\frac{A}{S_{LF}} = {{\left( {1/N^{2}} \right)\mspace{11mu} {\sum\limits_{i = 1}^{K}\left( {1/{c_{f}(i)}^{2}} \right)}} = {\left( {1/N^{2}} \right){\sum\limits_{i = 1}^{K}\left( \frac{d_{\max}(i)}{d_{\min}(i)} \right)^{2}}}}$

-   -   The sum Σ_(i=1) ^(K) (1/c_(f)(i)²) for constant layered scene        decomposition layers is monotonically decreasing and tending        towards 1.

Therefore, layered scene decomposition layers located closer to thedisplay plane achieve a lower compression ratio than layers of the samewidth located further away from the display plane. To maximizeefficiency, layered scene decomposition layers with a more narrow widthare located closer to the display plane, and wider layered scenedecomposition layers are located further away from the display plane;this placement maintains a uniform compression rate across the scene.

Number and Size of Layered Scene Decomposition Layers

To determine the number of layers and the size of layers required forthe layered scene decomposition, a light field display with an α(t)=tidentity function, is provided as an example. The consideration of thisidentity function is not intended to limit the scope or spirit of thepresent disclosure, as other functions can be utilized. The skilledtechnician in the field to which the invention pertains will appreciatethat while the display D=(M_(x), M_(y), N_(u), N_(v), f, a, D_(LP)) isdefined with a single identity function α, each light fieldplanar-parameterized pinhole projector within an array ofplanar-parameterized pinhole projectors may have a unique identityfunction α.

To losslessly represent fronto-planar surfaces (assuming no occlusions),a single layered scene decomposition layer with a front boundary locatedat depth Z_(DOF) represents the system from Z_(DOF) to infinity. Togenerate a core representation, layered scene decomposition layersbeyond the deepest layer located at the light field display's maximumdepth of field are not considered, as these layers do not provideadditional representative power from the core representationperspective; this applies to both the inner and outer frustum volumelayer sets.

Within the region from the display plane to the maximum depth of fieldof the display (for both the inner and outer frustum volume layer sets),the layered scene decomposition layers utilize maximum and minimumdistance depths that are integer multiples of the light field display fvalue. Layered scene decomposition layers with a more narrow widthprovide a better per-layer compression ratios, thereby providing betteroverall scene compression ratios. However, a greater number of layers inthe decomposition increases the amount of processing required fordecoding, as a greater number of layers must be reconstructed andmerged. The present disclosure accordingly teaches a layer distributionscheme with differential layer depths. In one embodiment, layered scenedecomposition layers (and by correlation the light fields represented bysaid layers) with a more narrow width are located closer to the displayplane, and the layer width (i.e., the depth difference between the frontand back layer boundaries) increases exponentially as the distance fromthe display plane increases.

Disparity Encoding/Decoding

The encoded layered scene decomposition representation of a light fieldproduced from a sampling scheme applied to each layer is principallycomprised of a plurality of pixels including RGB color and disparity.Generally speaking, selecting an appropriate bit width for the disparity(depth) field of the pixel is important, as the width of this fieldimproves the accuracy of the operation during reconstruction. However,the use of an increased number of bits contributes negatively to thecompression rate achieved.

In the present disclosure, each layer of RGB color and disparity pixelsspecified by the given sampling scheme has a specific range of disparitycorresponding to the individual pixels. The present disclosure exploitsthis narrow range of disparity within each layered scene decompositionlayer to increase the accuracy of the depth information. In conventionalpixel representations, the range of disparity for an entire scene ismapped to a fixed number of values. For example, in 10-bit disparityencoding, there can only be 1024 distinct depth values. In the layeredscene decomposition of the present disclosure, the same fixed number ofvalues are applied to each layered scene decomposition layer, as eachlayer has known depth boundaries. This is advantageous as thetransmission bandwidth can be reduced by decreasing the width of thedepth channel, while maintaining pixel reconstruction accuracy. Forexample, when the system implements a disparity width of 8-bits and thescene is decomposed into 8 layered scene decomposition layers, a totalof 2048 distinct disparity values can be used, with each layer having256 distinct possible values based on 8-bit representation. This is moreefficient than mapping the entire range of possible disparity valueswithin the inner or outer frustum to a given number of bits.

The present disclosure utilizes the same number of bits, but the bitsare interpreted and distinctly represent disparity within each layeredscene decomposition layer. Since each layered scene decomposition layeris independent from each other, depth (bit) encoding can differ for eachlayer and can be designed to provide a more accurate fixed pointrepresentation. For example, a layered scene decomposition layer closerto the display screen has smaller depth values and can use a fixed pointformat with a small number of integer bits and a large number offractional bits, while layered scene decomposition layers further awayfrom the display screen has larger depth values and can use a fixedpoint format with a large number of integer bits and a small number offractional bits.

The fractional bits are configurable on a per layer basis:

MinFixedPoint=1/(2^(FractionalBits))

MaxFixedPoint=2^(16-FractionalBits)−MinFixedPoint

Disparity is calculated from the depth in the light fieldpost-processing stage and encoded using the following formula:

ScaleFactor = (MaxFixedPoint − MinFixedPoint)/(NearClipDisparity − FarClipDisparity)EncodedDisparity = (Disparity − FarClipDisparity) ⋆ ScaleFactor + MinFixedPoint

Disparity is decoded using the following formula:

ScaleFactor = (MaxFixedPoint − MinFixedPoint )/(NearClipDisparity − FarClipDisparity )UnencodedDisparity = (EncodedDisparity − MinFixedPoint )/ScaleFactor + FarClipDisparity

Generalized and Illustrative Embodiment—CODEC Implementation andApplications Overview

The present disclosure defines an encoder-decoder for various types ofangular pixel parameterizations, such as, but not limited to, planarparameterizations, arbitrary display parameterizations, a combination ofparameterizations, or any other configuration or parameterization type.A generalized and illustrative embodiment of the present disclosureprovides a method to generate a synthetic light field formulti-dimensional video streaming, multi-dimensional interactive gaming,or other light-field display scenarios. A rendering system and processesare provided that can drive a light-field display with real-timeinteractive content. The light-field display does not require long-termstorage of light fields, however, the light fields must be rendered andtransmitted at low latency to support an interactive user experience.

FIG. 7 provides a CODEC system overview of the generalized, illustrativeembodiment of the present invention. A gaming engine or interactivegraphics computer (70) transmits three-dimensional scene data to GPU(71). The GPU encodes the data and sends it over the display port (72)to a decoding unit (73) containing a decoding processor such as an FPGAor ASIC. The decoding unit (73) sends decoded data to a light-fielddisplay (74).

FIG. 1 illustrates another generalized, exemplary layered scenedecomposition CODEC system, where light field data from a synthetic orvideo data source (50) is input to encoder (51). A GPU (43) encodes theinner frustum volume data, dividing it into a plurality of layers, andGPU (53) encodes the outer frustum volume data, dividing it into anadditional plurality of layers. While FIG. 1 illustrates separate GPUs(43, 53) dedicated for the inner and outer frustum volume layers, asingle GPU can be utilized to process both the inner and outer frustumvolume layers. Each of the layered scene decomposition layers aretransmitted to decoder (52), where the plurality of inner frustum volumelayers (44(1) through 44(*)) and the plurality of outer frustum volumelayers (54(1) through 54(*)) of a light field are decoded and mergedinto a single inner frustum volume layer (45) and a single outer frustumvolume layer (55). As per double frustum rendering, the inner and outerfrustum volumes are then synthesized (merged) into a single,reconstructed set of light field data (56), otherwise referred to hereinas a “final light field” or “display light field”.

FIGS. 10 to 13 illustrate exemplary CODEC process implementationsaccording to the present disclosure.

FIG. 10 illustrates an exemplary layered scene decomposition CODECmethod, whereby 3D scene data in the format of image description orlight field data is loaded to an encoder (400) for encoding, whereupondata (sub)sets as illustrated in the figure, or alternatively the entiredata set representing the 3D scene is partitioned (403). In the case ofthe identification of 3D scene data subsets for partitioning (402), itis understood that the identification process is a general process stepreference which is intended to simply refer to the ability to partitionthe data set in one pass, or in groupings (e.g. to encode inner frustrumand outer frustum data layers as illustrated in more detail in FIG. 11),as may be desired according to the circumstances. In this regard, theidentification of data subsets may imply pre-encoding processing stepsor processing steps also forming part of the encoding sub-process stage(401). Data subsets may be tagged, specified, confirmed, scanned andeven compiled or grouped at the time of partitioning to produce a set oflayers (decomposition of the 3D scene) (403). Following the partitioningof data subsets (403), each data layer is sampled and rendered accordingthe present disclosure to produce compressed (image) data (404).Following data layer compression the compressed data is transmitted to adecoder (405) for the decoding sub-process (406) comprisingdecompression, decoding and re-composition steps to (re)construct a setof light fields (407), otherwise referred to herein as “layered lightfields”, layered light field images and light field layers. Theconstructed layered light fields are merged to produce the final lightfield (408) displaying the 3D scene (409).

An exemplary, parallel CODEC process is illustrated in FIG. 13 foroptimizing the delivery of a light field representing a 3D scene inreal-time (e.g. to minimize artifacts). The process comprises the stepsof loading 3D scene data to an encoder (700), encoding and compressingthe residue encoded representation (701) of the final light field,transmitting the residue encoded representation (702) to a decoder,decoding the residue encoded representation and using the residueencoded representation with the core encoded representation to producethe final light field (703) and display the 3D scene at a display (704).

FIG. 11 illustrates an embodiment related to the embodiment shown inFIG. 10 in that two data (sub)sets (502, 503) derived based on the 3Dscene data (500) are identified for partitioning (501) and thepartitioning of each data set into layers of differential depths isimplemented according to two different layering schemes for each dataset (504, 505), i.e. equivalent to a plurality of data layers. Each set(plurality) of data layers (506, 507) representing an inner frustum andouter frustum volume of a light field display respectively aresubsequently sampled on a per layer basis according to sampling scheme(508, 509); and each sampled layer is rendered to compress the data andproduce two sets of compressed (image) data (510, 511) in process steps(508, 509), respectively. The sets of compressed data (510, 511)encoding the sets of light fields corresponding to the sets of datalayers (506, 507), are then combined (512) to produce a layered, coreencoded representation (513) of a final (display) light field.

FIG. 12 illustrates an embodiment of a CODEC method or process toreconstruct a set of light fields and produce a final light field at adisplay. The set of light fields (layered light fields) is(re)constructed from the core encoded representation (513) usingmulti-stage view synthesis protocols (600). A protocol (designated asVS1-VS8) is applied (601-608) to each of the eight layers of the coreencoded representation (513), which protocols may or may not bedifferent depending on characteristics of each data layer light field tobe decoded. Each protocol may apply a form of non-linear interpolationtermed herein as edge adaptive interpolation (609) to provide good imageresolution and sharpness in the set(s) of layered light fields (610)reconstructed from the core encoded representation of said fields ensureimage sharpness. The layered light fields (610) are merged, in this caseillustrating the merging of two sets of light fields (611, 612)corresponding to two data subsets to produce two sets of merged lightfields (613, 614). The merged sets of light fields (613, 614) mayrepresent, for example, the inner frustrum and outer frustum volumes ofa final light field and can be accordingly merged (615) to produce saidfinal light field (616) at a display.

CODEC Encoder/Encoding

Encoding according to the present disclosure is designed to support thegeneration of real-time interactive content (for example, for gaming orsimulation environments) as well as existing multi-dimensional datasetscaptured through light field generalized pinhole cameras or cameraarrays.

For a light field display D, a layered scene decomposition L, and asampling scheme S, the system encoder produces the elemental imagesassociated with the light fields corresponding to each layered scenedecomposition layer included in the sampling scheme. Each elementalimage corresponds to a generalized pinhole camera. The elemental imagesare sampled at the resolution specified by the sampling scheme and eachelemental image includes a depth map.

Achieving rendering performance to drive real-time interactive contentto multi-dimensional display with a significantly high resolution andsize presented significant challenges overcome with the application of ahybrid or combination rendering approach to resolve the deficiencies ofrelying solely on any one technique as described herein.

When given identity function α, the set of generalized pinhole camerasspecified by the encoding scheme for a given layered scene decompositionlayer can be systematically rendered using standard graphics viewportrendering. This rendering method results in a high number of draw calls,particularly for layered scene decomposition layers with samplingschemes including large numbers of the underlying elemental images.Therefore, in a system utilizing layered scene decomposition forrealistic, autostereoscopic light field displays, this rendering methodalone does not provide real-time performance.

A rendering technique utilizing standard graphics draw calls restrictsthe rendering of a generalized pinhole camera's planar parameterizations(identity function a) to perspective transformations. Hardware-optimizedrasterization functions provide the performance required forhigh-quality real-time rendering in traditional two-dimensionaldisplays. These accelerated hardware functions are based on planarparameterizations. Alternatively, parallel oblique projections canutilize standard rasterized graphics pipelines to render generalizedpinhole camera planar parameterizations.

The present disclosure contemplates the application of rasterization torender the generalized pinhole camera views by converting sets oftriangles into pixels on the display screen. When rendering largenumbers of views, every triangle must be rasterized in every view;oblique rendering reduces the number of rendering passes required foreach layered scene decomposition layer and can accommodate any arbitraryidentity function α. The system utilizes one parallel oblique projectionper angle specified by the identity function α. Once the data isrendered, the system executes a “slice and dice” block transform (seeU.S. Pat. Nos. 6,549,308 and 7,436,537) to re-group the stored data fromits by-angle grouping into an elemental image grouping. The “slice anddice” method alone is inefficient for real-time interactive contentrequiring many separate oblique rendering draw calls when a large numberof angles are to be rendered.

An arbitrary identity function α can also be accommodated by aray-tracing rendering system. In ray tracing, specifying arbitraryangles does not require higher performance than accepting planarparameterizations. However, for real-time interactive content requiringrendering systems utilizing the latest accelerated GPUs, rasterizationprovides more reliable performance scalability than ray tracingrendering systems.

The present disclosure provides several hybrid rendering approaches toefficiently encode a light field. In one embodiment, encoding schemesrender layered scene decomposition layers located closer to the displayplane, with more images requiring less angular samples, and layerslocated further away from the display plane, with less images and moreangular samples. In a related embodiment, perspective rendering, obliquerendering, and ray tracing are combined to render layered scenedecomposition layers; these rendering techniques can be implemented in avariety of interleaved rendering methods.

According to the generalized, illustrative embodiment of the disclosure,one or more light fields are encoded by a GPU rendering an array oftwo-dimensional pinhole cameras. The rendered representation is createdby computing the pixels from the sampling scheme applied to each of thelayered scene decomposition layers. A pixel shader performs the encodingalgorithm. Typical GPUs are optimized to produce a maximum of 2 to 4pinhole camera views per scene in one transmission frame. The presentdisclosure requires rendering hundreds or thousands of pinhole cameraviews simultaneously, thus multiple rendering techniques are employed torender data more efficiently.

In one optimized approach, the generalized pinhole cameras in thelayered scene decomposition layers located further away from the displayplane are rendered using standard graphics pipeline viewport operations,known as perspective rendering. The generalized pinhole cameras in thelayered scene decomposition layers located closer to the display planeare rendered using the “slice and dice” block transform.

Combining these methods provides high efficiency rendering for layeredplenoptic sampling theory sampling schemes. The present disclosureprovides layered scene decomposition layers wherein layers locatedfurther away from the display plane contain a smaller number ofelemental images with a higher resolution and layers located closer tothe display plane contain a greater number of elemental images with alower resolution. Rendering the smaller number of elemental images inthe layers further away from the display plane with perspectiverendering is efficient, as the method requires only a single draw callfor each elemental image. However, at some point, perspective renderingbecomes or is inefficient for layers located closer to the displayplane, as these layers contain a greater number of elemental images,requiring an increased number of draw calls. Since elemental imageslocated in layers located closer to the display plane correspond to arelatively small number of angles, oblique rendering can efficientlyrender these elemental images with a reduced number of draw calls. Inone embodiment a process to determine where the system should utilizeperspective rendering, oblique rendering, or ray tracing to render thelayered scene decomposition layers is provided, Applying a thresholdalgorithm, each layered scene decomposition layer is evaluated tocompare the number of elemental images to be rendered (i.e., the numberof perspective rendering draw calls) to the size of the elemental imagesrequired at the particular layer depth (i.e., the number of obliquerendering draw calls), and the system implements the rendering method(technique) requiring the least number of rendering draw calls.

Where standard graphics calls cannot be utilized, the system canimplement ray tracing instead of perspective or oblique rendering.Accordingly, in another embodiment, an alternative rendering methodrenders layers located closer to the display plane, or a portion of thelayers located closer to the display plane, using ray tracing.

In ray-tracing rendering systems, each pixel in a layered scenedecomposition layer is associated with a light ray defined by the lightfield. Each ray is cast and the intersection with the layered scenedecomposition is computed as per standard ray tracing methodologies. Raytracing is advantageous when rendering an identity function α which doesnot adhere to the standard planar parameterizations expected by thestandard GPU rendering pipeline, as ray tracing can accommodate thearbitrary ray angles that are challenging for traditional GPU rendering.

The skilled technician in the field to which the invention pertains willappreciate that there are multiple rendering methods and combinations ofrendering methods that can successfully encode the layered scenedecomposition elemental images. Other rendering methods may provideefficiency in different contexts, dependent upon the system's underlyingcomputational architecture, the utilized sampling scheme, and theidentity function α of the light field display.

CODEC Decoder/Decoding

Decoding according to the present disclosure is designed to exploit theencoding strategy (sampling and rendering). The core representation as aset of layered light fields from a downsampled layered scenedecomposition is decoded to reconstruct the light fields LF^(O) andLF^(P). Consider a display D=(M_(x), M_(y), N_(u), N_(v), f, a, D_(LP))with a layered scene decomposition L=(K₁, K₂, L^(O), L^(P)) and anassociated sampling scheme S=(M_(s), R). The elemental images aredecoded by reconstructing the light fields LF^(O) and LF^(P) fromdeconstructed LF^(O) and LF^(P) light fields downsampled as specified bysampling scheme S. The pixels align such that the inner and outerfrustum volume layers located closer to the display plane are reviewedfirst, moving to inner and outer frustum volume layers located furtheraway from the display plane until a non-empty pixel is located, and thedata from the non-empty pixel is transmitted to the empty pixel closerto the display plane. In an alternative embodiment, particularimplementations may restrict viewing to the inner frustum volume or theouter frustum volume of the light field display, thereby requiring thedecoding of one of LF^(O) or LF^(P).

In one embodiment, a decoding process is represented by the followingpseudocode:

Core Layered Decoding:

for each I_(i)ϵL^(o):

ReconLF(LF_(l) _(i) ,D_(m)[LF_(l) _(i) ],S)

LF ^(O) =LF _(l) _(i) _(*m) LF _(l) _(i−1) // or LF _(l) _(i) _(*m) LF_(l) _(i+1) (front-back vs. back-front)

A similar procedure reconstructs LF^(P). Each layered scenedecomposition layer is reconstructed from the limited samples defined bythe given sampling scheme S. Each of the inner frustum volume layers orthe outer frustum volume layers are merged to reproduce LF^(O) orLF^(P).

ReconLF can be executed in various forms with varying computational andpost-CODEC image quality properties. The ReconLF input is the subset ofLF_(l) _(i) data defined by the given sampling scheme S and thecorresponding downsampled depth map D_(m)[LF_(l) _(i) ]. Depth-ImageBased Rendering (DIBR), as described by Graziosi et al., can reconstructthe input light field. DIBR can be classified as a projection renderingmethod. In contrast to re-projection techniques, ray-casting methods,such as the screen space ray casting taught by Widmer et al., canreconstruct the light fields. Ray casting enables greater flexibilitythan re-projection, but increases computational resource requirements.

In the DIBR approach, elemental images specified in the sampling schemeS are used as reference “views'” to synthesize the missing elementalimages from the light field. As described by Vincent Jantet in “LayeredDepth Images for Multi-View Coding” and by Graziosi et al., when thesystem uses DIBR reconstruction, the process typically includes forwardwarping, merging, and back projection.

Application of the back projection technique avoids producing cracks andsampling artifacts in synthesized views such as elemental images. Backprojection assumes that the elemental image's depth map or disparity mapis synthesized along with the necessary reference images required toreconstruct the target image; such synthesis usually occurs through aforward warping process. With the disparity value for each pixel in thetarget image, the system warps the pixel to a corresponding location ina reference image; typically this reference image location is notaligned on the integer pixel grid, so a value from the neighboring pixelvalues must be interpolated. Implementations of backprojection known inthe art use simple linear interpolation. Linear interpolation, however,can be problematic. If the warped reference image location sits on ornear an object edge boundary, the interpolated value can exhibitsignificant artifacts, as information from across the edge boundary isincluded in the interpolation operation. The synthesized image isgenerated with a “smeared′” or blurred edge.

The present disclosure provides a back projection technique for theinterpolation substep, producing a high quality synthesized imagewithout smeared or blurred edges. The present disclosure introducesedge-adaptive interpolation (EAI), where the system incorporates depthmap information to identify the pixels required by the interpolationoperation to calculate the colour of the warped pixels in a referenceimage. EAI is a nonlinear interpolation procedure that adapts andpreserves edges during low-pass filtering operations. Consider a displayD=(M_(x), M_(y), N_(u), N_(v), f, a, D_(LP)) with a target imageI_(t)(x, y), a reference image I_(r)(x, y), and depth maps D_(m)(I_(t))and D_(m)(I_(r)). The present disclosure utilizes the depth mapD_(m)(I_(t)) pinhole camera parameters (f, α, etc.) and the relativeposition of the display's array of planar-parameterized pinholeprojectors to warp each I_(t) pixel integer (x, y,) to a real-numberposition (x_(w), y_(w)) in I_(r). In the likely scenario where (x_(w),y_(w)) is not located on an integer coordinate position, a value must bereconstructed based on I_(r) integer samples.

Linear interpolation methods known in the art reconstruct I_(r)(x_(w),y_(w)) from the four nearest integer coordinates located in a 2×2 pixelneighborhood. Alternate reconstruction methods use larger neighborhoods(such as 3×3 pixel neighborhoods), generating similar results withvarying reconstruction quality (see Marschner et al., “An evaluation ofreconstruction filters for volume rendering”). These linearinterpolation methods have no knowledge of the underlying geometry ofthe signal. The smeared or blurred edge images occur when thereconstruction utilizes pixel neighbors belonging to different objects,separated by an edge in the images. The erroneous inclusion of colourfrom other objects creates ghosting artifacts. The present disclosureremedies this reconstruction issue by providing a method to weigh oromit pixel neighbors by using the depth map D_(m)(I_(r)) to predict theexistence of edges created when a plurality of objects overlap.

FIG. 3 illustrates textures (80,83), where a sampling location,illustrated as a black dot, is backprojected into another image beingreconstructed. The sampling location is located near the boundary of adark object with a white background. In a first reconstruction matrix(81), the full 2×2 neighborhood reconstructs the sampling location valueusing a known technique such as linear interpolation. This results in anon-white pixel (82), as the dark object is included in thereconstruction. The second reconstruction matrix (84) uses the EAItechnique of the present disclosure, reconstructing the samplinglocation from the three neighboring white pixels. EAI detects the objectedge and omits the dark pixel, resulting in the correct white pixelreconstruction (85).

For a fixed, arbitrary coordinate (x_(r), y_(r)) in the target imageI_(t)(x, y), d_(t) defines the location depth:

d _(t) =D _(m)[I _(r)(x _(r) ,y _(r))]

The target image coordinate (x_(r), y_(r)) warps to the reference imagecoordinate (x_(w), y_(w)).

For an m-sized neighborhood of points close to (x_(w), y_(w)), the setN_(S)={(x_(i), y_(i))|1≤i≤m}. The weight for each of the neighbors isdefined as:

w _(i) =f(d _(t) ,D _(m)[I _(r)](x _(i) ,y _(i))]

Where w_(i) is a function of the depth (x_(r), y_(r)) and the depth ofthe neighbor of (x_(w), y_(w)) corresponding to index i. The followingequation represents an effective w_(i) for a given threshold t_(e):

$w_{i} = \left\{ \begin{matrix}{{1{{d_{t} - {{D_{m}\left\lbrack I_{r} \right\rbrack}\left( {x_{i},y_{i}} \right)}}}} < t_{e}} \\{{0{{d_{t} - {{D_{m}\left\lbrack I_{r} \right\rbrack}\left( {x_{i},y_{i}} \right)}}}} \geq t_{e}}\end{matrix} \right.$

The threshold t_(e) is a feature size parameter. The weight functiondetermines how to reconstruct I_(r) (x_(r), y_(r)):

I _(r)(x _(r) ,y _(r))=Recon(w ₁ I _(r)(x ₁ ,y ₁),(w ₂ I ₂(x ₂ ,y ₂), .. . (w _(m) I _(r)(x _(m) ,y _(m)))

The Recon function can be a simple modified linear interpolation, wherethe w_(i) weights are incorporated with standard weighting proceduresand re-normalized to maintain a total weight of 1.

The present disclosure also provides a performance-optimized decodingmethod for reconstructing the layered scene decomposition. Consider adisplay D=(M_(x), M_(y), N_(u), N_(v), f, a, D_(LP)) with a layeredscene decomposition L=(K₁, K₂, L^(O), L^(P)) and an associated samplingscheme S=(M_(s), R). The elemental images are decoded by reconstructingthe light fields LF^(O) and LF^(P) from deconstructed LF^(O) and LF^(P)light fields downsampled as specified by sampling scheme S. As notedabove, particular implementations may restrict viewing to the innerfrustum volume or the outer frustum volume of the light field display,thereby requiring the decoding of one of LF^(O) or LF^(P).

LF^(O) can be reconstructed by decoding the elemental images specifiedby sampling scheme S. The ReconLF method for particular layers does notinclude inherent constraints regarding the order that the missing pixelsof the missing elemental images are to be reconstructed. It is an objectof the present disclosure to reconstruct missing pixels using a methodthat maximizes throughput; a light field large enough for an effectivelight field display requires an exceptional amount of data throughput toprovide content at an interactive frame rate, therefore improvedreconstruction data transmission is required.

The present disclosure introduces a basic set of constraints to improvepixel reconstruction with improved data transmission for content at aninteractive frame rate. Consider a single light field L_(i)ϵL_(o),containing M_(x)×M_(y) elemental images, as input to ReconLF. The pixels(in other words, the elemental images) are reconstructed in two basicpasses. Each pass operates in separate dimensions of the array ofelemental images; the system executes the first pass as a columndecoding, and the second pass as a row decoding, to reconstruct each ofthe pixels. While the present disclosure describes a system employingcolumn decoding followed by row decoding, this is not meant to limit thescope and spirit of the invention, as a system employing row decodingfollowed by column decoding can also be utilized.

In the first pass, the elemental images specified by sampling scheme Sare used as reference pixels to fill in missing pixels. FIG. 4illustrates the elemental images in the matrix as B, or blue pixels. Themissing pixels are synthesized strictly from reference pixels in thesame column. These newly synthesized column-wise pixels are shown as R,or red pixels, in FIG. 5. Newly reconstructed pixels written to a bufferand act as further pixel references for the second pass, whichreconstructs pixels reference pixels located in the same row as otherelemental images. FIG. 6 illustrates these newly synthesized row-wisepixels as G, or green pixels.

In one embodiment a process for reconstructing a pixel array isrepresented by the following pseudocode algorithm:

-   -   Dimensional Decomposition Light Field Reconstruction:    -   Pass 1:    -   for each row of elemental images in L_(i)        -   for each missing elemental image in the row        -   for each row in elemental image            -   load (cache) pixels from same row in reference images        -   for each pixel in missing row            -   reconstruct pixel from reference information and write    -   Pass 2:    -   for each column of elemental images in L_(i)        -   for each missing elemental image in the column        -   for each column in elemental image            -   load (cache) reference pixels from same column        -   for each pixel in missing column            -   reconstruct pixel from reference information and write

This performance-optimized decoding method allows the row-decoding andcolumn-decoding constraints to limit the effective working data setrequired for reconstruction operations.

To reconstruct a single row of a missing elemental image, the systemonly requires the corresponding row of pixels from the referenceelemental images. Likewise, to reconstruct a single column of a missingelemental image, the system only requires the corresponding column ofpixels from the reference elemental images. This method requires asmaller dataset, as decoding methods previously known in the art requireentire elemental images for decoding.

Even when decoding relatively large elemental image sizes, the reduceddataset can be stored in a buffer while rows and columns of missingelemental images are being reconstructed, thereby providing improveddata transmission.

Once all the rendered data has been decoded, and each of the pluralityof inner and outer display volume layers are reconstructed, the layersare merged into a single inner display volume layer and a single outerdisplay volume layer. The layered scene decomposition layers can bepartially decompressed in a staged decompression, or can be fullydecompressed simultaneously. Algorithmically, the layered scenedecomposition layers can be decompressed through a front-to-back orback-to-front process. The final double frustum merging process combinesthe inner and outer display volume layers to create the final lightfield for the light field display.

To gain a better understanding of the invention described herein, thefollowing examples are set forth with reference to the Figures. It willbe understood that these examples are intended to describe illustrativeembodiments of the invention and are not intended to limit the scope ofthe invention in any way.

EXAMPLES Example 1: Exemplary Encoder and Encoding Method for a LightField Display

The following illustrative embodiment of the invention is not intendedto limit the scope of the invention as described and claimed herein, asthe invention can successfully implement a plurality of systemparameters. As described above, a conventional display as previouslyknown in the art consists of spatial pixels substantially evenly-spacedand organized in a two-dimensional row, allowing for an idealizeduniform sampling. By contrast, a three-dimensional (3D) display requiresboth spatial and angular samples. While the spatial sampling of atypical three-dimensional display remains uniform, the angular samplescannot necessarily be considered uniform in terms of the display'sfootprint in angular space.

In the illustrative embodiment, a plurality of light fieldplanar-parameterized pinhole projectors provide angular samples, alsoknown as directional components of the light field. The light fielddisplay is designed for a 640×480 spatial resolution and a 512×512angular resolution. The plurality of planar-parameterized pinholeprojectors are idealized with identity function α. The pitch betweeneach of the plurality of planar-parameterized pinhole projectors is 1mm, thereby defining a 640 mm×400 mm display screen. The display has a120° FOV, corresponding to an approximate focal length f=289 μm.

This light field display contains 640×480×512×512=80.5 billion RGBpixels. Each RGB pixel requires 8 bits, therefore one frame of the lightfield display requires 80.5 billion×8×3=1.93 Tb. For a light fielddisplay providing interactive content, data is driven at 30 frames/s,requiring a bandwidth of 1.93 Tb×30 frames/s=58.0 Tb/s. Current displaysknown in the art are driven by DisplayPort technology providing maximumbandwidths of 32.4 Gb/s, therefore such displays would require over 1024DisplayPort cables to provide the tremendous bandwidth required byinteractive light field displays, resulting in cost and form-factordesign constraints.

The illustrative embodiment delivers data to a light field display froma computer equipped with an accelerated GPU with dual DisplayPort 1.3cables output. We consider a conservative maximum throughput of 40 Gb/s.The encoded frames must be small enough for transmission over theDisplayPort connection to a decoding unit physically located closer tothe light field display.

The layered scene decomposition of the illustrative embodiment isdesigned to allow the required data throughput. With the dimensionsdefined above, the maximum depth of field of the light field display isZ_(DOF)=(289 microns)(512)=147968 microns=147.986 mm. The layered scenedecomposition places a plurality of layered scene decomposition layerswithin the depth of field region of the light field display, ensuringthat the distance of the layered scene decomposition layers from thedisplay screen is less than Z_(DOF). This illustrative example describesa light field display with objects located only within the inner frustumvolumes of the display. This illustrative example is not intended tolimit the scope of the invention, as the invention can successfullyimplement a plurality of system parameters, such as a light fielddisplay with objects located only within the outer frustum volume of thedisplay, or a light field display with objects located within both theinner and outer frustum volumes of the display; embodiments limited toone frustum volume require a smaller number of layered scenedecomposition layers, thereby marginally decreasing the size of theencoded light field to be produced.

The illustrative embodiment defines ten layered scene decompositionlayers. When necessary, additional layered scene decomposition layerscan be added to capture data that could be lost to occlusions, or toincrease the overall compression rate. However, additional layered scenedecomposition layers require additional computation from the decoder,thus the number of layered scene decomposition layers is carefullychosen. The illustrative embodiment specifies the ten layered scenedecomposition layers from their front and back boundaries and assumesthat the dividing planes of the layer are parallel to the display plane.

Each layered scene decomposition layer is located at a defined distancefrom the display plane, where the distances are specified in terms ofmultiples of focal length f, up to the maximum depth of field of 512 f.Layered scene decomposition layers with a more narrow width areconcentrated closer to the display plane, and the layer width (i.e., thedepth difference between the front and back layer boundaries) increasesexponentially by powers of 2 as the distance from the display planeincreases. This embodiment of the invention is not intended to limit thescope of the invention, as other layer configurations can be implementedsuccessfully.

The following table (Table 1) describes the layered scene decompositionlayer configurations of the illustrative embodiment, and provides asampling scheme based on plenoptic sampling theory to create sub-sampledlayered scene decomposition layers:

TABLE 1 Maximum distance between Total data size sampled required (24Elemental elemental Elemental bit color, 8 bits Front Back image imagesimages for depth/ Layer boundary boundary resolution (sampling gap)sampled disparity) 0 1f 1f 1 × 1 0 640 × 480  7.37 Mbits 1 1f 2f 2 × 2 1321 × 241  9.90 Mbits 2 2f 4f 4 × 4 2 214 × 161 17.64 Mbits 3 4f 8f 8 ×8 4 161 × 97  31.98 Mbits 4 8f 16f  16 × 16 8 72 × 55 32.44 Mbits 5 16f 32f  32 × 32 16 41 × 31 41.65 Mbits 6 32f  64f  64 × 64 32 21 × 16 44.04Mbits 7 64f  128f  128 × 128 64 11 × 9  51.90 Mbits 8 128f  256f  256 ×256 128 6 × 5 62.91 Mbits 9 256f  512f  512 × 512 256 4 × 3 100.66Mbits  Total: 400.49 Mbits 

In the above table, layer 0 captures images that are to be displayed atthe display screen plane, as in a conventional two-dimensional displayknown in the art. Layer 0 contains 640×480 pixels at a fixed depth, soit does not require any depth information. The total data size iscalculated for each pixel with an RGB value and a depth value for 8 bitseach (alternate embodiments may require larger bit values, such as 16bits). In the illustrative embodiment, the elemental image resolutionand sampling gap are calculated from the formulas described above, andthe sampling scheme chosen reflects the elemental image resolution andsampling gap restrictions.

As described in the above table, the combined layered scenedecomposition system has a total size of 400.5 Mb. Therefore, to producedata at a rate of 30 frames/s, a bandwidth of 30×0.4005=12.01 GB/s isrequired. This encoded form is sent over the dual DisplayPort 1.3cables, along with additional information required to represent sceneocclusions.

In the illustrative embodiment, the layered scene decomposition layersare configured by an encoder, efficiently implementing an obliquerendering technique to produce the layers located closer to the displayplane (layers 0 to 5) and a perspective rendering technique to producethe layers located further away from the display plane (layers 6 to 9).Each elemental image corresponds to a single rendering view.

At layer 6, the number of separate angles to be rendered (64×64=4096)exceeds the number of views to be rendered (21×16=336); this signals thetransition in efficiency between the oblique and perspective renderingmethods. It should be noted that specific implementation aspects mayprovide additional overhead that skews the exact optimal transitionpoint. For use with modern graphics acceleration techniques known in theart, perspective rendering can be efficiently implemented using geometryshader instancing. Multiple views are rendered from the same set ofinput scene geometry without repeatedly accessing the geometry throughdraw calls and without repeatedly accessing memory to retrieve the exactsame data.

FIG. 8 illustrates the illustrative embodiment, with ten layered scenedecomposition layers (100-109) in the inner frustum volume (110). Theinner frustum volume layers extend from the display screen plane (300).The layers are defined as described in the table above, for example, thefront boundaries of the inner frustum volume layer 0 (100) is 1f, innerfrustum volume layer 1 (101) is 1f, inner frustum volume layer 2 (102)is 2f, inner frustum volume layer 3 (103) is 4f, and so on. Innerfrustum volume layers 0 to 5 (100-105) and outer frustum volume layers 0to 5 (200-205) are rendered with the oblique rendering technique, andouter frustum volume layers 6 to 9 (106-109) are rendered with theperspective rendering technique.

FIG. 9 illustrates an alternate embodiment, with ten layered scenedecomposition layers (100-109) in the inner frustum volume (110) and tenlayered scene decomposition layers (200-209) in the outer frustum volume(210). The inner and outer frustum volume layers extend from the displayscreen plane (300). While the inner and outer frustum volume layers areillustrated as mirror images from each other, the inner and outerfrustum volume may have differing numbers of layers, layers of differentsizes, or layers of different depths. Inner frustum volume layers 0 to 5(100-105) and outer frustum volume layers 0 to 5 (200-205) are renderedwith the oblique rendering technique, and outer frustum volume layers 6to 9 (106-109) and outer frustum volume layers 6 to 9 (206-209) arerendered with the perspective rendering technique.

An alternate embodiment can implement the system with a ray-tracingencoding based approach. Rendering a complete layered scenedecomposition layer representation can require increased GPUperformance, even with the optimizations described herein, as GPUs areoptimized for interactive graphics on conventional two-dimensionaldisplays where accelerated rendering of single views is desirable. Thecomputational cost of the ray-tracing approach is a direct function ofthe number of pixels the system is to render. While the layered scenedecomposition layer system contains a comparable number of pixels tosome two-dimensional single view systems, the form and arrangement ofsaid pixels differs greatly due to layer decomposition and correspondingsampling schemes. Therefore, there may be implementations where tracingsome or all of the rays is a more efficient implementation.

Example 2: CODEC Decoder and Decoding Method for a Light Field Display

In the illustrative embodiment of the invention, the decoder receivesthe 12.01 GB/s of encoded core representation data, plus any residuerepresentation data, from the GPU over dual DisplayPort 1.3 cables. Thecompressed core representation data is decoded using a customized FPGA,ASIC, or other integrated circuit to implement efficient decoding(residue representation data is decoded separately, as illustrated inFIG. 13). The 12.01 GB/s core representation is decompressed to 58 Tb/sfor the final light field display. Note that this core representationdoes not include the residue representations necessary to renderocclusions. The

$\frac{58\mspace{14mu} {{Tb}/s}}{12.01\mspace{14mu} {{GB}/s}}$

provides a compression ratio of 4833:1; while this is a high performancecompression ratio, the reconstructed light field data may still exhibitocclusion-based artifacts unless residue representation data is includedin the reconstruction.

For the illustrative embodiment shown in FIG. 8, data is decoded byreconstructing individual layered scene decomposition layers and mergingthe reconstructed layers into an inner frustum volume layer. For analternate embodiment, such as illustrated in FIG. 9, the data is decodedby reconstructing individual layered scene decomposition layers andmerging the reconstructed layers into an inner frustum volume layer andan outer frustum volume layer.

A single layered scene decomposition layer can be reconstructed fromgiven sampling scheme sampling of data using view synthesis techniquesfrom the field of Image-Based Rendering which are known in the art. Forexample, Graziosi et al. specify using reference elemental images toreconstruct the light field in a single pass. This method uses referenceelemental images offset from the reconstructed image in multipledimensions. Because the elemental image data represents threedimensional scene points (including RGB color and disparity), pixels aredecoded as a nonlinear function (although fixed on the directionalvector between the reference and target elemental images), thereforerequiring a storage buffer of equal size to the decoding referenceelemental images. When decoding larger elemental images, this can creatememory storage or bandwidth constraints, depending on the decodinghardware.

For a light field display with an elemental image size of 512×512 pixelswith 24-bit color, a decoder requires a buffer capable of storing512×512=262,144 24-bit values (without disparity bits in this example).Current high-performance FPGA devices provide internal block memory(BRAM) organized as 18/20-bit wide memory and 1024 memory locationswhich can be used as a 36/40-bit wide memory with 512 memory locations.A buffer capable of reading and writing an image in the same clock cycleis large enough to hold two reference elemental images, as the nonlineardecoding process causes the write port to use a non-deterministic accesspattern. Implementing this buffer in an FPGA device for a 512×512 pixelimage requires 1024 BRAM blocks. Depending on the reconstructionalgorithm used, multiple buffers may be required in each decoderpipeline. To meet the data rate of a high-density light-field display,the system may require more than one hundred parallel pipelines, whichis magnitudes more pipelines than current FPGA devices. Because eachbuffer requires an independent read/write port, it may not be possibleto implement such a system on current ASIC devices.

The present disclosure circumvents buffer and memory limitations bydividing the pixel reconstruction process into multiple,single-dimension stages. The present disclosure implements onedimensional reconstruction to fix the directional vector between thereference elemental images and the target to a rectified path. Whilereconstruction remains nonlinear, the reference pixel to be translatedto the target location is locked to the same row or column location ofthe target pixel. Therefore, decoder buffers only need to capture onerow or one column at a time. For the elemental image of 512×512 pixelswith 24-bit color described above, the decoder buffer is organized as a24-bit wide, 1024 deep memory requiring two 36/40×512 BRAM. Therefore,the present disclosure has reduced the memory footprint by a factor of512, or multiple magnitudes. This allows a display pixel fill raterequiring over a hundred decoding pipelines to be supported by currentFPGA devices.

Multi-stage decoding architectures require two stages to reconstruct thetwo dimensional pixel array in a light-field display. The two stages areorthogonal to one another and reconstruct rows or columns of elementalimages. The first decoding stage may require a pixel scheduler to ensurethat output pixels ordered to be compatible with the next stage inputpixels. Due to the extremely high bandwidth required by each decodingstage, some output pixels from a previous stage may need to be reused toreduce local storage requirements. In this case, an external buffer canbe used to capture all of the output pixels from a first stage so thesubsequent decoding stage can efficiently access pixel data, reducinglogic resources and memory bandwidth.

The present disclosure's multi-stage decoding with an external memorybuffer allows the decoding process to transfer the required memorybandwidth from expensive on-die memory to lower cost memory devices suchas double data rate (DDR) memory devices. A high performance decodingpixel scheduler ensures maximum reference pixel reuse from this externalmemory buffer, allowing the system to use narrower or slower memoryinterfaces.

The disclosures of all patents, patent applications, publications anddatabase entries referenced in this specification are herebyspecifically incorporated by reference in their entirety to the sameextent as if each such individual patent, patent application,publication and database entry were specifically and individuallyindicated to be incorporated by reference.

Although the invention has been described with reference to certainspecific embodiments, various modifications thereof will be apparent tothose skilled in the art without departing from the spirit and scope ofthe invention. All such modifications as would be apparent to oneskilled in the art are intended to be included within the scope of thefollowing claims.

REFERENCE LIST

-   BANKS, MARTIN S., DAVID M. HOFFMAN, JOOHWAN KIM AND GORDON    WETZSTEIN. “3D Displays” Annual Review of Vision Science. 2016. pp.    397-435.-   CHAI, JIN-XIANG, XIN TONG, SHING-CHOW CHAN, AND HEUNG-YEUNG SHUM.    “Plenoptic Sampling”-   CLARK, JAMES J., MATTHEW R. PALMER AND PETER D. LAWRENCE. “A    Transformation Method for the Reconstruction of Functions from    Nonuniformly Spaced Samples” IEEE Transactions on Acoustics, Speech,    and Signal Processing. October 1985. pp 1151-1165. Vol. ASSP-33, No.    4.-   DO, MINH N., DAVY MARCHAND-MAILLET AND MARTIN VETTERLI. “On the    Bandwidth of the Plenoptic Function” IEEE Transactions on Image    Processing. pp. 1-9-   GORTLER, STEVEN J., RADEK GRZESZCZUK, RICHARD SZELISKI, AND    MICHAEL F. COHEN. “The Lumigraph” 43-52.-   HALLE, MICHAEL W. AND ADAM B. KROPP. “Fast Computer Graphics    Rendering for Full Parallax Spatial Displays”-   JANTET, VINCENT. “Layered Depth Images for Multi-View Coding”    Multimedia. pp. 1-135. Universite Rennes 1, 2012. English.-   LEVOY, MARC, AND PAT HANRAHAN. “Light Field Rendering” SIGGRAPH. pp.    1-12.-   MARSCHNER, STEPHEN R. AND RICHARD J. LOBB. “An Evaluation of    Reconstruction Filters for Volume Rendering” IEEE Visualization    Conference 1994.-   PIAO, YAN, AND XIAOYUAN YAN. “Sub-sampling Elemental Images for    Integral Imaging Compression” IEEE. pp. 1164-1168. 2010.-   VETRO, ANTHONY, THOMAS WIEGAND, AND GARY J. SULLIVAN. “Overview of    the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4    AVC Standard.” Proceedings of the IEEE. pp. 626-642. April 2011.    Vol. 99, No. 4.-   WIDMER, S., D. PAJAK, A. SCHULZ, K. PULLI, J. KAUTZ, M. GOESELE,    AND D. LUEBKE. “An Adaptive Acceleration Structure for Screen-Space    Ray Tracing”-   ZWICKER, M., W. MATUSIK, F. DURAND, H. PFISTER. “Antialiasing for    Automultiscopic 3D Displays” Eurographics Symposium on Rendering.    2006.

What is claimed is:
 1. A scene decomposition method comprising:receiving light field data from a data source, the light field datacomprising inner frustum volume data and outer frustum volume dataseparated by a display surface; partitioning the inner frustum volumedata and the outer frustum volume data into a plurality of scenedecomposition layers comprising a plurality of inner frustum volumelayers and a plurality of outer frustum volume layers; and decoding andmerging the plurality of inner frustum volume layers and the pluralityof outer frustum volume layers into a single reconstructed set of lightfield data to reconstruct a display light field.
 2. The method of claim1, further comprising decoding and merging the plurality of innerfrustum volume layers and the plurality of outer frustum volume layersinto a single inner frustum volume layer and a single outer frustumvolume layer to reconstruct a display light field.
 3. The method ofclaim 1, wherein the inner frustum volume data is encoded by a first agraphical processing unit and the outer frustum volume data is encodedby a second graphical processing unit.
 4. The method of claim 1, whereinthe inner frustum volume data and the outer frustum volume data isencoded by one or more graphical processing units.
 5. The method ofclaim 1, wherein the data source is a synthetic data source, a videodata source, a 3D description of a scene, or a combination thereof. 6.The method of claim 1, wherein the method is carried out in real-time.7. The method of claim 1, wherein the volume layers closer to thedisplay surface have smaller depth values and the volume layers furtheraway from the display screen have larger depth values.
 8. The method ofclaim 1, further comprising identifying redundancies in the light fielddata and discarding elemental images based on observation that elementalimages representing neighboring points in space contain overlappedinformation.
 9. The method of claim 1, wherein the inner frustum volumelayers and the outer frustum volume layers increase in depth as thedistance between the volume layer and the display surface increases. 10.The method of claim 1, wherein the depth of the inner frustum volumelayers and outer frustum volume layers increases exponentially relativeto the distance between the volume layer and the display surface. 11.The method of claim 1, further comprising reconstructing a pixel arrayfrom reference elemental images used to construct the light fieldcorresponding to each volume layer.
 12. The method of claim 1, whereinpartitioning the inner frustum volume data and the outer frustum volumedata has a scene decomposition layer spanning the display surface.
 13. Asystem for generating a layered scene decomposition comprising: aplurality of orthoscopic cameras for capturing an inner frustum volumelight field, the inner frustum volume light field partitioned into aplurality of inner frustum scene decomposition layers; a plurality ofpseudoscopic cameras for capturing an outer frustum volume light field,the outer frustum volume light field partitioned into a plurality ofouter frustum scene decomposition layers; and at least one graphicalprocessing unit for processing the inner frustum volume light field andthe outer frustum volume light field, wherein each of the inner frustumscene decomposition layers and outer frustum scene decomposition layershas an associated spatial, angular, and depth value.
 14. The system ofclaim 13, wherein the at least one graphical processing unit comprises afirst graphical processing unit for processing the inner frustum volumelight field image and a second graphical processing unit for processingthe outer frustum volume light field image.
 15. The system of claim 13,comprising a plurality of graphical processing units.
 16. The system ofclaim 13, wherein the plurality of orthoscopic cameras and the pluralityof pseudoscopic cameras are arranged in a plurality of pinhole camerapairs, each pinhole camera pair comprising one of the plurality oforthoscopic cameras and one of the plurality of pseudoscopic cameras.17. The system of claim 16, wherein the plurality of pinhole camerapairs are parameterized using a warping function.
 18. The system ofclaim 17, wherein each of the plurality of pinhole camera pairs uses aunique warping function.
 19. A computer-implemented method comprising:receiving light field data from a data source, the light field datacomprising multi-dimensional scene data; generating a layered scenedecomposition from the light field data comprising a plurality of datalayers each at a depth relative to a display surface, the depth of eachdata layer increasing as the distance between the data layer and thedisplay surface increases; and reconstructing the layered scenedecomposition into a single light field to provide a compressed layeredrepresentation of the multi-dimensional scene data.
 20. The method ofclaim 19, wherein the steps of generating a layered scene decompositionand reconstructing the layered scene decomposition are done inreal-time.
 21. The method of claim 19, wherein the light field data fromthe data source has a total data size greater than the compressedlayered representation.
 22. The method of claim 19, wherein theplurality of data layers comprise inner frustum data layers and outerfrustum data layers.